Response code for Chinese spiders? [closed]

Posted by pt2ph8 on Server Fault See other posts from Server Fault or by pt2ph8
Published on 2012-10-30T16:44:53Z Indexed on 2012/10/30 17:04 UTC
Read the original article Hit count: 157

Filed under:

blocking

|

spider

My server is being "attacked" by Chinese spiders that don't respect the rules in my robots.txt. They are being very aggressive and using a lot of resources, so I'm going to set up some rules in nginx to block them by user agent.

Question: which response code should I return, 403, 444 (empty response in nginx) or something else? I'm wondering how the spiders will react to different status codes. What's the best practice?

© Server Fault or respective owner

Related posts about blocking

Non-Blocking I/O Made Possible in Java

as seen on Internet.com - Search for 'Internet.com'
Java SE7 "Dolphin" release is nearing and we're chomping at the bit. So let's dig in and review non-blocking IO, a feature of java.nio (New I/O) package that is a part of Java v1.4, v1.5 and v1.6 and we'll also take a peek at the java.nio.file (NIO.2) package. >>> More
Non-Blocking I/O Made Possible in Java

as seen on Internet.com - Search for 'Internet.com'
Java SE7 "Dolphin" release is nearing and we're chomping at the bit. So let's dig in and review non-blocking IO, a feature of java.nio (New I/O) package that is a part of Java v1.4, v1.5 and v1.6 and we'll also take a peek at the java.nio.file (NIO.2) package. >>> More
Generate a proper 404 page for blocked sites via /etc/hosts instead of redirect to localhost

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I have blocked some websites by editing /etc/hosts and adding several newline-entries in the following manner: 0.0.0.0 www.domain.com And it works. The only thing is: when a website is visited which is blocked, the browser is redirected to my http://localhost, resulting in a directory listing or… >>> More
Block a website on HTTPS [closed]

as seen on Programmers - Search for 'Programmers'
I would like to block some websites on their HTTPS version and allow them on HTTP. The main websites involved are Youtube and Google Images/Videos. This is because on the HTTP version, I can enforce the Safesearch filter on those platforms, whereas I cannot on the HTTPS version. For me, this is a… >>> More
What should we tell our unsupported IE6 users?

as seen on Programmers - Search for 'Programmers'
In the upcoming version of our web app, we've broken IE6, and we don't intend to fix it. We've had a clear warning posted for IE6 users for some months; we've decided it's time not to support it. My question is: how should we communicate this to our users? Some people here feel that we should block… >>> More

Related posts about spider

Creating a spider using Scrapy, Spider generation error.

as seen on Stack Overflow - Search for 'Stack Overflow'
I just downloaded Scrapy (web crawler) on Windows 32 and have just created a new project folder using the "scrapy-ctl.py startproject dmoz" command in dos. I then proceeded to created the first spider using the command: scrapy-ctl.py genspider myspider myspdier-domain.com but it did not work and… >>> More
Site crawler/spider that tosses results into mysql

as seen on Server Fault - Search for 'Server Fault'
It's been suggested that we use mysql for our site's search as it'd be running on the same server that hosts our web server (nginx) and our db (mysql). Since not all of our pages are created from the database, it's been suggested that we have a crawler that can crawl the site, and toss the page url… >>> More
Getting Started with Python: Attribute Error

as seen on Stack Overflow - Search for 'Stack Overflow'
I am new to python and just downloaded it today. I am using it to work on a web spider, so to test it out and make sure everything was working, I downloaded a sample code. Unfortunately, it does not work and gives me the error: "AttributeError: 'MyShell' object has no attribute 'loaded' " I am… >>> More
How to create a web crawler/spider/robot?

as seen on Stack Overflow - Search for 'Stack Overflow'
Is there a way to make a web robot like websiteoutlook.com does? I need something that searches the internet for URLs only...I don't need links, descriptions, etc. What is the best way to do this without getting too technical? I guess it could even be a cronjob that runs a PHP script grabbing URLs… >>> More
How do I block a user-agent from Apache

as seen on Pro Webmasters - Search for 'Pro Webmasters'
How do I realize a UA string block by regular expression in the config files of my Apache webserver? For example: if I would like to block out all bots from Apache on my debian server, that have the regular expression /\b\w+[Bb]ot\b/ or /Spider/ in their user-agent. Those bots should not be able… >>> More